Raise Public Scrutiny

This position paper lists reasons why the results in software maintenance are so diicult to assess and use, proposes three possible countermeasures what and why, and shortly discusses the roadblocks. Note: This paper is written in a rather apodictic tone, but that is for brevity only. Why we have a mire of evidence When one dives into the literature on software maintenance , the amount of suggestions and experience reports is overwhelming, the validity of the statements is often dubious but hard to assess, commonalities between results are diicult to understand and characterize , and contradictions between diierent reports which occur frequently are diicult to resolve. The collective evidence is a morass; it is no wonder that so few organizations exploit research results successfully. These eeects are due to the following problems: Overall volume. The total number of reports on maintenance is quite large. In principle this is positive, but it also has two drawbacks: Gaining and maintaining a representative overview of the literature is dif-cult and the quality of some of the contributions is low. Variability. There is a multitude of rather diierent maintenance contexts and tasks, which makes understanding commonalities diicult. This is an intrinsic problem, which can possibly be controlled but not be resolved. Incompleteness. Important information is often missing in reports, in particular with respect to the context of the results, but also about the results themselves or about their meaning. Incomparability. As a result of the lack of information it is often almost impossible to relate the results from diierent reports even if they could be comparable in principle. Missing baselines. It is frequently impossible to understand data in reports because it is free-oating: without any reliable baseline or control group against which to judge it. Ambiguity. Reports are often ambiguous because some of their important terms have multiple possible deenitions e.g. LOC, module, hour, design, development , testing but no deenition is provided. Overall incompleteness. Even if one had a complete overview of the literature, the resulting picture would be distorted, because reports on failures are misleadingly rare. Mixing report and interpretation. In many reports it is not at all clear which parts refer to objective facts measured, counted, etc. and which refer to estimations or subjective judgements. Possible solutions I conjecture that the following three countermeasures, if implemented to a suucient degree, would much reduce the problems mentioned above. Clariied terminology. We need …